Testing and evaluation of functional correctness and communications performance of distributed applications over Mobile Ad hoc NETworks (MANETs) pose significant challenges in both feasibility and scalability. With respect to feasibility, ad hoc network hardware is often experimental and available only in small quantities, so conducting evaluation of more than a few hardware devices is either infeasible or impractical for cost reasons. With respect to scalability, running large-scale evaluations of MANETs in a physical terrain is very costly in terms of logistics, so the number of tests that can be performed would typically be much less than adequate.
In order to address the aforementioned challenges, virtual MANETs based on simulated or emulated networks are used for application testing and evaluation. Applications send packets over virtual MANETs where their network characteristics including packet delivery latency, effective link bandwidth, packet delivery jitter, etc., are similar to those of real MANETs. In most cases, a virtual network built using precise simulation models can provide the desired level of approximation to address fidelity concerns.
Network aware applications are a special class of applications that interact with a network not only by using the network for communication purposes, but also configure or read the status of network devices. The present invention provides the ability to automatically construct standard APIs for accessing the information residing in a simulated or emulated network, and instantiate these APIs.
Most prior work in network testbeds does not deal with network-aware applications. Most network simulators offer basic mechanisms for exchanging information between the simulation model and outside world. Such mechanisms do not offer standardized APIs network aware applications use to interact with networks. SNT's EXata, described in Scalable Network Technologies, “EXata,” 2009. [Online], available at http://www.scalable-networks.com/products/exata.php, offers small subset of standardized APIs for accessing some parameters, but offers no automatic tools for deriving such APIs to match specific applications. Their solution is not scalable and offers a limited means for accessing driver's parameters through piggybacking information on packets. The method is incomplete, not general, and does not offer automation.
Prior work applicable to functional testing and performance evaluation of distributed software systems can be roughly divided into the following categories by the characteristics of the approaches:
Specialized network testbeds: This type of approaches addresses testing a network consisting of some specific hardware and/or software components (e.g., radios and/or OSs). Some approaches allow the use of real distributed systems for testing (e.g., D. Johnson, T. Stack, R. Fish, D. M. Flickingery, L. Stoller, R. Ricci, and J. Lepreau, “Mobile Emulab: A Robotic Wireless and Sensor Network Testbed.” in Proc. of IEEE INFOCOM'06, Barcelona, Spain, 2006, while others only allow models for a specific network simulator to be used as test targets e.g., P. De, A. Raniwala, S. Sharma, and T.-C. Chiueh, “MiNT: A Miniatur-ized Network Testbed for Mobile Wireless Research,” in Proc. of IEEE INFOCOM'05, Miami, Fla., 2005). The VAN testbed approach of the present invention aims to be as generic as possible with respect to simulators, models, and OSs/applications.
Full network emulation: An emulated network, from the viewpoint of packet forwarding, introduces various conditions, e.g., packet loss, delay, jitter. When a network is “emulated”, both the packet forwarding process and network protocol interactions are neither real nor being modeled as stepwise transitions using finite state machines reflecting actual protocols. Often, protocol stack interactions are modeled by programs offering simple, abstracted behaviors, e.g., a packet entering the emulated network is either regarded as dropped along the route, or delivered to its destination node after certain delay, with or without error. To use emulated network for testing and evaluation of real distributed software, packets generated by real software must be “injected” into the emulated network as if routed into a real network. M. Carson and D. Santay, “NIST Net: a linux-based network emulation tool,” ACM SIGCOMM Computer Communication Review, vol. 33:3, pp. 111-126, 2003, A. Vandat, K. Yocum, K. Walsh, P. Mahadevan, D. Kostic, J. Chase, and D. Becker, “Scalability and accuracy in a large-scale network emulator,” in Proc. of USENIX OSDI'02, Boston, Mass., 2002, B. White, J. Lepreau, L. Stoller, R. Ricci, S. Guruprasad, M. Newbold, M. Hibler, C. Barb, and A. Joglekar, “An integrated experimental environment for distributed systems and networks,” in Proc. of USENIX OSDI'02, Boston, Mass., 2002, Y. Zhang and W. Li, “An integrated environment for testing mobile ad-hoc networks,” in Proc. of ACM MOBIHOC'02, Lausanne, Switzerland, 2002, P. Zheng and L. M. Ni, “Empower: A network emulator for wireline and wireless networks,” in Proc. of IEEE INFOCOM'03, San Francisco, Calif., 2003, and J. Zhou, Z. Ji, and R. Bagrodia, “TWINE: A Hybrid Emulation Testbed for Wireless Networks and Applications,” in Proc. of IEEE INFO-COM'06, Barcelona, Spain, 2006 provide examples of network emulation designs. A high level of abstraction in full network emulation approaches typically restricts their use with network-aware applications.
Hybrid network simulation: These types of approaches split the network protocol stack simulation onto two different machines and provide a mechanism that creates a virtual network that behaves identically to a real network from the perspective of real applications. A good example is J. Zhou, Z. Ji, M. Takai, and R. Bagrodia, “MAYA: Integrating hybrid network modeling to the physical world,” ACM Transactions on Modeling and Computer Simulation, vol. 14(2), 2004.
More recently, significant work has been done on complete testbed technologies that allow a user to combine generic network emulation/hybrid simulation with real OS and application code, most notably in Emulab Team, “Emulab: total network testbed,” Flux Group, School of Computing, University of Utah, 2009. [Online] available at: http://www.emulab.net and Network and Communication Systems Branch, “MANE publications,” Naval Research Laboratory, 2008. [Online]. Available: http://cs.itd.nrl.navy.mil/pubs/. Neither includes support for network-aware applications. Limited support for setting parameters of the lower layers of a split stack is provided in T. Braun, T. Staub, and R. Gantenbein, “VirtualMesh: An Emulation Framework for Wireless Mesh Networks in OMNeT++,” in Proc. of OMNeT++2009 (SIMUTools 2009), 2009; however, their approach is lazy (depends on data sent), insufficiently general (addresses only writing parameter values), has high overhead (piggybacks on every data packet), and does not automate instrumentation of arbitrary simulation models.
Several existing mature and evolving network emulation tools include some level of support for software-in-the-loop or system-in-the-loop emulation, particularly ns-2 Team, “The Network Simulator—ns-2,” 2009. [Online]. Available at: http://www.isi.edu/nsnam/ns/ and ns-3 Team, “The ns-3 network simulator,” 2009. [Online]. Available at: http://www.nsnam.org/
Typically, the transparent IP packet forwarding functionality is exposed via an Ethernet-like network interface, and the user is expected to set up routing into the emulated network. Similarly to the complete testbeds mentioned above, emulation is restricted to real time, and no user-friendly facility is provided for applications to interact with the part of the node network stack implemented in the emulation. Scalable Network Technologies, “EXata,” 2009. [Online]. Available: http://www.scalable-networks.com/products/exata.php provides simulated SNMP agents that execute in the simulator, as part of the lower-layer simulated stack. These SNMP agents are able to respond to a small subset of SNMP requests generated from outside of the simulation. Although this approach provides basic support for some applications, it cannot cope with applications that require full SNMP support, e.g., OpenNMS Team, “OpenNMS: Enterprise-grade Open-source Network Management,” 2009. [Online]. Available: http://www.opennms.org/. Furthermore, execution of all the SNMP agents corresponding to all the nodes in a single centralized process (i.e. thesimulator) does not scale well, given that the computational and memory footprint of a full SNMP agent is non-negligible.
Note that all existing emulation tools do expose lower-level APIs for interacting with the emulated stack components, and the current VAN testbed functionality is built on top of such APIs for OPNET. However, the low-level APIs are very specific to a tool, require a considerable effort to learn and configure for the required fidelity, and do not, by themselves, provide a complete bridge between an unmodified network-aware application and the emulated stacks useful to a tester.
While this approach satisfies the testing and evaluation needs for applications that are agnostic of network activities and configurations, other applications, which are categorized as network-aware applications, need to interact with the network and/or respond to network configuration changes. Prominent examples of network-aware applications include:
i) network management applications that monitor network resources and/or configure network settings, and ii) communication middleware that adapts its communication strategies based on the observed network dynamics. Such applications are becoming more prevalent as MANET applications need to change their behavior according to the state of the network.
As a result, to enable the execution of network-aware applications over virtual networks, there is a need to allow applications to interact with the simulation/emulation at runtime in order to retrieve information from, and change configuration settings of, the virtual networks. Furthermore, virtual networks should expose these monitoring and control capabilities via a standard-based interface so that applications under test can run over virtual networks as they do over real networks, without any code modification. Traditionally, testbeds based on virtual networks do not support testing and evaluation of network-aware applications; existing testbeds that provide some form of interaction with the virtual network are limited by some of the following: approach generality, approach fidelity, and requirements on customizing applications under test.
Current solutions do not generate an external module agent for managing the local stack parameters implemented in the simulation/emulation. Such generic agent offers full functionality for accessing the emulated stack and thus can be used for any model and parameter.